Image sensor with image receiver and automatic image combining

ABSTRACT

Provided are an image sensor with one or more image receivers for image combining, and an imaging system and method therefor. The image sensor includes an image sensor array to generate first image data for a first image; a receiver to receive, into the image sensor, second image data for a second image; an image combination circuit coupled to the image sensor array and the receiver to receive the first image data and the second image data and combine the first image data and the second image data into combined image data for a single combined image, according to one or more image combination criteria, and at least one of the first image data and the second image data; and a transmitter coupled to the image combination circuit to transmit the combined image data for the combined image from the image sensor.

TECHNICAL FIELD

The present disclosure generally relates to the field of solid-stateimage sensor arrays, particularly to solid-state image sensor arrayshaving receivers for receiving additional images.

BACKGROUND

Digital cameras, scanners, and other imaging devices often use imagesensors, such as charge-coupled device (CCD) image sensors orcomplementary metal-oxide-semiconductor (CMOS) image sensors, to convertoptical signals to electrical signals for storage, processing, display,and the like. Recently, imaging devices have begun to include multipleimage sensors, for example to capture images using multiple spectra,multiple subjects, and the like.

SUMMARY

In general, one aspect disclosed features an imaging system comprising:an image sensor comprising a first image sensor array to generate firstimage data for a first image, a receiver to receive, into the imagesensor, second image data for a second image, an image combinationcircuit coupled to the first image sensor array and the receiver toreceive the first image data and the second image data and combine thefirst image data and the second image data into combined image data fora single combined image, according to one or more image combinationcriteria, and at least one of the first image data and the second imagedata, and a transmitter coupled to the image combination circuit totransmit the combined image data for the combined image from the imagesensor; and a second image sensor array coupled to the image sensor togenerate the second image data for the second image.

Embodiments of the imaging system may include one or more of thefollowing features. Some embodiments comprise an image signal processorto process the selected one of the first image data and the second imagedata into a processed image. Some embodiments comprise a display todisplay the processed image. In some embodiments, the first imageincludes a scene; the second image includes the scene; and the combinedimage is a high dynamic range image. In some embodiments, the firstimage includes a subject; the second image includes the subject; and thecombined image is a three-dimensional (3D) image. In some embodiments,the first image and the second image are captured substantiallyconcurrently. In some embodiments, the image sensor, and an image sensorcapturing the second image, possess different image capturecharacteristics, wherein the image capture characteristics include atleast one of: pixel size; sensitivity; read noise; gain; exposure time;and spectrum.

In general, one aspect disclosed features an image sensor comprising: animage sensor array to generate first image data for a first image; areceiver to receive, into the image sensor, second image data for asecond image; an image combination circuit coupled to the image sensorarray and the receiver to receive the first image data and the secondimage data and combine the first image data and the second image datainto combined image data for a single combined image, according to oneor more image combination criteria, and at least one of the first imagedata and the second image data; and a transmitter coupled to the imagecombination circuit to transmit the combined image data for the combinedimage from the image sensor.

Embodiments of the image sensor may include one or more of the followingfeatures. In some embodiments, the first image includes a scene; thesecond image includes the scene; and the combined image is a highdynamic range image. In some embodiments, the first image includes asubject; the second image includes the subject; and the combined imageis a three-dimensional (3D) image. In some embodiments, the first imageand the second image are captured substantially concurrently. In someembodiments, the image sensor, and an image sensor capturing the secondimage, possess different image capture characteristics, wherein theimage capture characteristics include at least one of: pixel size;sensitivity; read noise; gain; exposure time; and spectrum. In someembodiments, the receiver is a Mobile Industry Processor Interface(MIPI) receiver; and the transmitter is a MIPI transmitter. Someembodiments comprise a single integrated circuit comprising the imagesensor.

In general, one aspect disclosed features a method for image sensor, themethod comprising: generating, in an image sensor array of the imagesensor, first image data for a first image, responsive to receivinglight; receiving into the image sensor, second image data for a secondimage; combining the first image data and the second image data intocombined image data for a single combined image, according to one ormore image combination criteria, and at least one of the first imagedata and the second image data; and transmitting the combined image datafor the combined image from the image sensor.

Embodiments of the method may include one or more of the followingfeatures. In some embodiments, the first image includes a scene; thesecond image includes the scene; and the combined image is a highdynamic range image. In some embodiments, the first image includes asubject; the second image includes the subject; and the combined imageis a three-dimensional (3D) image. In some embodiments, the first imageand the second image are captured substantially concurrently. In someembodiments, the image sensor, and an image sensor capturing the secondimage, possess different image capture characteristics, wherein theimage capture characteristics include at least one of: pixel size;sensitivity; read noise; gain; exposure time; and spectrum. In someembodiments, the receiver is a Mobile Industry Processor Interface(MIPI) receiver; and the transmitter is a MIPI transmitter.

BRIEF DRAWINGS DESCRIPTION

The present disclosure describes various embodiments that may beunderstood and fully appreciated in conjunction with the followingdrawings:

FIG. 1 illustrates a conventional imaging system using multiple imagesensor arrays.

FIG. 2 illustrates an imaging system that includes an image sensorhaving an image data receiver and automatic image combining according tothe technology disclosed herein.

FIG. 3 illustrates two image sensor arrays having the same opticalformat but different image capture characteristics.

FIG. 4 illustrates a process that may be performed by the imaging systemof FIG. 2 according to embodiments of the disclosed technology.

FIG. 5 illustrates a process for the image combination according toembodiments of the disclosed technology.

DETAILED DESCRIPTION

The present disclosure describes embodiments with reference to thedrawing figures listed above. Persons of ordinary skill in the art willappreciate that the description and figures illustrate rather than limitthe disclosure and that, in general, the figures are not drawn to scalefor clarity of presentation. Such skilled persons will also realize thatmany more embodiments are possible by applying the inventive principlescontained herein and that such embodiments fall within the scope of thedisclosure which is not to be limited except by the claims.

FIG. 1 illustrates a conventional imaging system using multiple imagesensor arrays. Referring to FIG. 1, the imaging system 100 includes aplurality of image sensor arrays 102 a through 102 n, and an imageprocessor 104. Each image sensor array 102 generates image dataresponsive to receiving light 110, for example such as light reflectedby, or radiated by, a subject. The image processor 104 includes an imagesignal processor 106 for processing the image data generated by theimage sensor arrays 102. In the conventional imaging system 100 of FIG.1, the image processor 104 must include multiple receivers (RX) 108 athrough 108 n to receive the image data from the multiple image sensorarrays 102 a through 102 n. In particular, the image processor 104 mustinclude a separate receiver 108 for each image sensor array 102.

FIG. 2 illustrates an imaging system that includes an image sensorhaving an image data receiver and automatic image combining according tothe technology disclosed herein. Referring to FIG. 2, the imaging system200 includes an image sensor 214, an image processor 204, a display 216,and one or more external image sensor arrays 212 a through 212 n. Theexternal image sensor arrays 212 a through 212 n are referred to hereinas “external” because they are external to the image sensor 214. Theimage processor 204 includes an image signal processor 206, and a singlereceiver 228. The image sensor 214 includes an internal image sensorarray 202, one or more receivers 208 a through 208 n, a singletransmitter (TX) 218, an image combination circuit 222, and imagecombination criteria 220. The image sensor 214 may be fabricated as asingle integrated circuit. The image sensor arrays 202, 212 may befabricated using any suitable technology. For example, the image sensorarrays 202, 212 may be fabricated as arrays of CMOS CCDs. The imagesensor arrays 202, 212 may all be fabricated using the same technology,or using different technologies. In some embodiments, each receiver 208,228 is a Mobile Industry Processor Interface (MIPI) receiver, andtransmitter 218 is a MIPI transmitter. That is, receivers 208, 228 andtransmitter 218 operate according to a MIPI specification. But in otherembodiments, receivers 208, 228 and transmitter 218 may operateaccording to other specifications.

In some embodiments, while the image sensor arrays 202, 212 may have thesame optical format, one or more of the image sensor arrays 202, 212 maypossess different image capture characteristics. These image capturecharacteristics may include, for example, pixel size, sensitivity, readnoise, gain, exposure time, spectrum, other image capturecharacteristics, any combination thereof, and the like. FIG. 3illustrates two image sensor arrays having the same optical format butdifferent image capture characteristics. Referring to FIG. 3, two imagesensor arrays 302, 304 are illustrated. Both image sensor arrays 302,304 have the same optical format. That is, both image sensor arrays 302,304 have the same size and shape. However, the image sensor arrays 302,304 have different pixel sizes. In particular, the pixels of the imagesensor array 304 are four times as large as the pixels of the imagesensor array 302.

By having multiple image sensor arrays 202, 212 with different imagecapture characteristics, multiple concurrent images of a subject orscene having different qualities may be combined into a single imagethat is preferable to any of its constituent images. For example,combining images of a scene captured using differing gain and exposuretimes may yield a combined image having extended bit depth and extendeddynamic range. Such combinations minimize motion artifacts, and deliversuperior image quality in the most demanding and difficult lightingconditions.

Various embodiments may be employed to combine concurrent images of asubject from different viewpoints, and to combine the images to create athree-dimensional (3D) image. Such a 3D image may be used with 3Dprinting techniques to reproduce the subject, at the same size or atdifferent scales.

In the example of FIG. 3, the image sensor array 304 having larger pixelsize will possess higher sensitivity, and less read noise, than theimage sensor array 302 having smaller pixel size. Accordingly, for darkscenes, the image produced by the image sensor array 304 having largerpixel sizes will have a higher quality than the image produced by theimage sensor array 302 having smaller pixel size. The image sensor array302 having smaller pixel size will have higher resolution and a largecharge handling capacity, meaning the pixels are not as easilysaturated. The image sensor array 302 having smaller pixel sizetherefore will produce a higher quality image for a well lit scene thanthe image produced by the image sensor array 304 having larger pixelsizes. To further optimize these image sensor arrays 302, 304 for lightand dark scenes, the image sensor array 302 having smaller pixel sizemay be implemented with a longer exposure, while the image sensor array304 having larger pixel sizes may be implemented with a shorterexposure.

Referring again to FIG. 2, responsive to receiving light 210, theinternal image sensor array 202 may generate image data, and may becoupled to the image combination circuit 222 to provide that generatedimage data to the image combination circuit 222. Similarly, eachexternal image sensor array 212 may generate image data responsive toreceiving the light 210. Each external image sensor array 212 mayprovide its image data to a corresponding receiver 208 within the imagesensor 214. The receivers 208 a through 208 n may receive the image datainto the image sensor 214, and may be coupled to the image combinationcircuit 222 to provide that received image data to the image combinationcircuit 222. In accordance with the image combination criteria 220, theimage combination circuit 222 may combine two or more of the imagescaptured by the image sensor arrays 202, 212. The image combinationcircuit 222 may be coupled to the transmitter 218 to provide the imagedata for the combined image to the transmitter 218, which may transmitthe image data from the image sensor 214.

The receiver 228 of the image processor 204 may receive the data for thecombined image into the image processor 204, and may provide the imagedata to the image signal processor 206. The image signal processor 206may process the image data according to any image processing algorithm.For example, the image signal processor may process the image data toimplement enhancements such as tone mapping, and the like. The imageprocessor 204 may provide the processed image data to the display 216,which may display the processed combined image in accordance with theprocessed image data.

Embodiments of the disclosed technology feature several advantages.Compared with the conventional implementation of FIG. 1, the disclosedtechnology, for example as shown in FIG. 2, requires only one receiverto be implemented within the image processor 204. Fewer receivers meansfewer data lines, fewer clock lines, and the like, thereby decreasingthe complexity and cost of the image processor 204. Furthermore, theimage combination function implemented by the image combination circuit222 is now performed outside the image processor 204, further reducingthe complexity and cost of the image processor 204. Using image sensorarrays 202, 212 configured to capture lights of differing spectrafeatures advantages as well. For example, an infrared image sensor arraymay produce an image showing structural features that differ from thosein an image produced by a visible light image sensor array. The imagecombination circuit 222 may combine the images to produce an imagehaving useful features from both constituent images, for example forfacial recognition in a high security environment.

FIG. 4 illustrates a process that may be performed by the imaging system200 of FIG. 2 according to embodiments of the disclosed technology.Although elements of the process are depicted in a certain sequence, inother embodiments the elements may be performed in other sequences,concurrently, or any combination thereof. In some embodiments, one ormore elements may be omitted. Referring to FIG. 4, the process 400 mayinclude generating, in the internal image sensor array 202 of the imagesensor 214, first image data for a first image, responsive to receivingthe light 210, at 402.

The process 400 may include receiving, into the image sensor 214, secondimage data for a second image, at 404. The second image data may begenerated by one or more of the external image sensor arrays 212. Theinternal and external image sensor arrays 202, 212 may capture imagessubstantially concurrently. For example, each captured image maycorrespond to a frame in a sequence of frames, and the image sensorarrays 202, 212 may have the same frame rate and phase. The images maybe of the same subject, or of different subjects. The images may be fromthe same viewpoint, or from different viewpoints. The images may includethe same scene, or different scenes.

The process 400 may include combining the first image data and thesecond image data according to one or more image combination criteria220, and at least one of the first image data in the second image data,at 406. The image combination criteria 220 may include, for example,extended dynamic range, extended bit depth, any combination thereof, andthe like. Process 400 may include transmitting the image data for thecombined image data from the image sensor 214, at 408.

FIG. 5 illustrates a process for the image combination according toembodiments of the disclosed technology. Although elements of theprocess are depicted in a certain sequence, in other embodiments theelements may be performed in other sequences, concurrently, or anycombination thereof. In some embodiments, one or more elements may beomitted. The process may be employed, for example by the imagecombination circuit 222 of FIG. 2. Referring to FIG. 5, the process 500begins with image data received from the internal image sensor array202, at 502, and image data received from one or more external imagesensor arrays 212, at 504. The process 500 may include an imageregistration process, at 506. During the image registration process, afeature detection process is executed to detect common features in thereceived image data. The common features are then used to size and alignthe images, for example through rotation, translation, expansion,contraction, and the like.

The process 500 may include a geometric correction process, at 508. Thegeometric correction process digitally manipulates the image data, forexample to ensure that each images projection precisely matches aspecific projection surface or shape. The geometric correction processmay compensate for artifacts such as distortion and the like.

The process 500 may include image combination process, at 510. Duringthe image combination process, the image data from the internal sensorarray 202, and the image data from one or more external image sensorarrays 212, which may be registered and/or corrected as described above,is combined create combined image data for a single combined image, at512. The combined image data may then be used, for example as describedabove.

It will also be appreciated by persons of ordinary skill in the art thatthe present disclosure is not limited to what has been particularlyshown and described hereinabove. Rather, the scope of the presentdisclosure includes both combinations and sub-combinations of thevarious features described hereinabove as well as modifications andvariations which would occur to such skilled persons upon reading theforegoing description. Thus the disclosure is limited only by theappended claims.

The invention claimed is:
 1. An imaging system comprising: an imagesensor comprising a first one of N image sensor arrays to generate firstimage data for a first image, wherein N is an integer greater than two,N−1 receivers each to receive, into the image sensor, second image datafor N−1 respective second images, wherein the first image and the N−1second images are captured concurrently; an image combination circuitcoupled to the first one of the image sensor arrays and the N−1receivers to receive the first image data and the second image data andcombine the first image data and the second image data into combinedimage data for a single combined image, according to one or more imagecombination criteria, and at least one of the first image data and thesecond image data, and a transmitter coupled to the image combinationcircuit to transmit the combined image data for the combined image fromthe image sensor; and N−1 second ones of the N image sensor arrays, eachcoupled to a respective one of the N−1 receivers of the image sensor togenerate the second image data for N−1 second images, wherein the N−1second ones of the N image sensor arrays are external to the imagesensor.
 2. The imaging system of claim 1, further comprising: an imagesignal processor to process of the first image data and the second imagedata into a processed image.
 3. The imaging system of claim 2, furthercomprising: a display to display the processed image.
 4. The imagingsystem of claim 1, wherein: the first image includes a scene; the N−1second images include the scene; and the combined image is a highdynamic range image.
 5. The imaging system of claim 1, wherein: thefirst image includes a subject; the N−1 second images include thesubject; and the combined image is a three-dimensional (3D) image. 6.The imaging system of claim 1, wherein the first one of the image sensorarrays, and the N−1 second ones of the image sensor arrays possessdifferent image capture characteristics, wherein the image capturecharacteristics include at least one of: pixel size; sensitivity; readnoise; gain; exposure time; and spectrum.
 7. An image sensor comprising:a first image sensor array to generate first image data for a firstimage; N−1 receivers to receive, into the image sensor, from N−1 secondimage sensor arrays externa to the image sensor, second image data forN−1 second images, wherein N is an integer greater than two, wherein thefirst image and the N−1 second images are captured concurrently; animage combination circuit coupled to the first image sensor array andthe N−1 receivers to receive the first image data and the second imagedata and combine the first image data and the second image data intocombined image data for a single combined image, according to one ormore image combination criteria, and at least one of the first imagedata and the second image data; and a transmitter coupled to the imagecombination circuit to transmit the combined image data for the combinedimage from the image sensor.
 8. The image sensor of claim 7, wherein:the first image includes a scene; the N−1 second images include thescene; and the combined image is a high dynamic range image.
 9. Theimage sensor of claim 7, wherein: the first image includes a subject;the N−1 second images include the subject; and the combined image is athree-dimensional (3D) image.
 10. The image sensor of claim 7, whereinthe first image sensor array and the N−1 second image sensor arrayspossess different image capture characteristics, wherein the imagecapture characteristics include at least one of: pixel size;sensitivity; read noise; gain; exposure time; and spectrum.
 11. Theimage sensor of claim 7, wherein: the receiver is a Mobile IndustryProcessor Interface (MIPI) receiver; and the transmitter is a MIPItransmitter.
 12. A single integrated circuit comprising the image sensorof claim
 7. 13. A method for a system comprising an image sensor and Nimage sensor arrays, wherein N is an integer greater than two, and theimage sensor includes a first one of the N image sensor arrays, themethod comprising: generating, in the first one of the N image sensorarrays of the image sensor, first image data for a first image,responsive to receiving light, wherein the first one of the image sensorarrays is internal to the image sensor; receiving into the image sensor,from N−1 second ones of the image sensor arrays, second image data forN−1 second images, wherein the N−1 second ones of the image sensorarrays are external to the image sensor, wherein the first image and theN−1 second images are captured concurrently; combining the first imagedata and the second image data into combined image data for a singlecombined image, according to one or more image combination criteria, andat least one of the first image data and the second image data; andtransmitting the combined image data for the combined image from theimage sensor.
 14. The method of claim 13, wherein: the first imageincludes a scene; the N−1 second images include the scene; and thecombined image is a high dynamic range image.
 15. The method of claim13, wherein: the first image includes a subject; the N−1 second imagesinclude the subject; and the combined image is a three-dimensional (3D)image.
 16. The method of claim 13, wherein the first one of the imagesensor arrays, and the N−1 second ones of the image sensor arrayspossess different image capture characteristics, wherein the imagecapture characteristics include at least one of: pixel size;sensitivity; read noise; gain; exposure time; and spectrum.
 17. Themethod of claim 13, wherein: the receiver is a Mobile Industry ProcessorInterface (MIPI) receiver; and the transmitter is a MIPI transmitter.